{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4be5d807",
   "metadata": {},
   "source": [
    "# Quickstart: Agentic retrieval in Azure AI Search\n",
    "\n",
    "Use this notebook to get started with [agentic retrieval](https://learn.microsoft.com/azure/search/search-agentic-retrieval-concept) in Azure AI Search, which integrates an Azure OpenAI chat completion model to process queries, retrieve relevant content from indexed documents, and generate natural-language answers.\n",
    "\n",
    "Steps in this notebook include:\n",
    "\n",
    "1. Creating and loading an `earth-at-night` search index.\n",
    "\n",
    "1. Creating an `earth-knowledge-source` that targets your index.\n",
    "\n",
    "1. Creating an `earth-knowledge-agent` that targets your knowledge source and an LLM for query planning and answer synthesis.\n",
    "\n",
    "1. Using the agent to fetch, rank, and synthesize relevant information from the index.\n",
    "\n",
    "This notebook provides a high-level demonstration of agentic retrieval. For more detailed guidance, see [Quickstart: Run agentic retrieval in Azure AI Search](https://learn.microsoft.com/azure/search/search-get-started-agentic-retrieval)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6712b97d",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "+ An [Azure AI Search service](https://learn.microsoft.com/azure/search/search-create-service-portal) on the Basic tier or higher with [semantic ranker enabled](https://learn.microsoft.com/azure/search/semantic-how-to-enable-disable).\n",
    "\n",
    "+ An [Azure AI Foundry project](https://learn.microsoft.com/azure/ai-foundry/how-to/create-projects) and Azure AI Foundry resource. When you create a project, the resource is automatically created.\n",
    "\n",
    "+ A [supported chat completion model](https://learn.microsoft.com/azure/search/search-agentic-retrieval-how-to-create#supported-models). This sample uses `gpt-4.1-mini`.\n",
    "\n",
    "+ A text embedding model. This sample uses `text-embedding-3-large`.\n",
    "\n",
    "+ [Visual Studio Code](https://code.visualstudio.com/download) with the [Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python) and [Jupyter package](https://pypi.org/project/jupyter/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f5fbd46",
   "metadata": {},
   "source": [
    "## Configure access\n",
    "\n",
    "This notebook assumes that you're using Microsoft Entra ID for authentication and role assignments for authorization.\n",
    "\n",
    "To configure role-based access:\n",
    "\n",
    "1. Sign in to the [Azure portal](https://portal.azure.com).\n",
    "\n",
    "1. On your Azure AI Search service:\n",
    "\n",
    "    1. [Enable role-based access](https://learn.microsoft.com/azure/search/search-security-enable-roles).\n",
    "    \n",
    "    1. [Create a system-assigned managed identity](https://learn.microsoft.com/azure/search/search-howto-managed-identities-data-sources#create-a-system-managed-identity).\n",
    "    \n",
    "    1. [Assign the following roles](https://learn.microsoft.com/azure/search/search-security-rbac#how-to-assign-roles-in-the-azure-portal) to yourself.\n",
    "    \n",
    "       + **Search Service Contributor**\n",
    "    \n",
    "       + **Search Index Data Contributor**\n",
    "    \n",
    "       + **Search Index Data Reader**\n",
    "\n",
    "1. On your Azure AI Foundry resource, assign **Cognitive Services User** to the managed identity of your search service."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "733bf308",
   "metadata": {},
   "source": [
    "## Set up connections\n",
    "\n",
    "The `sample.env` file contains environment variables for connections to Azure AI Search and Azure OpenAI in Azure AI Foundry. Agentic retrieval requires these connections for document retrieval, query planning, and query execution.\n",
    "\n",
    "To set up the connections:\n",
    "\n",
    "1. Sign in to the [Azure portal](https://portal.azure.com).\n",
    "\n",
    "1. Get the endpoints for Azure AI Search (`https://your-search-service.search.windows.net`) and Azure OpenAI in Azure AI Foundry (`https://your-foundry-resource.openai.azure.com`).\n",
    "\n",
    "1. Save the `sample.env` file as `.env` on your local system.\n",
    "\n",
    "1. Update the `.env` file with the retrieved endpoints."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34a54a0f",
   "metadata": {},
   "source": [
    "## Create a virtual environment\n",
    "\n",
    "The `requirements.txt` file contains the dependencies for this notebook. You can use a virtual environment to install these dependencies in isolation.\n",
    "\n",
    "To create a virtual environment:\n",
    "\n",
    "1. In Visual Studio Code, open the folder that contains `quickstart-agentic-retrieval.ipynb`.\n",
    "\n",
    "1. Press **Ctrl**+**Shift**+**P** to open the command palette.\n",
    "\n",
    "1. Search for **Python: Create Environment**, and then select **Venv**.\n",
    "\n",
    "1. Select a Python installation. We tested this notebook on Python 3.13.7.\n",
    "\n",
    "1. Select `requirements.txt` for the dependencies.\n",
    "\n",
    "Creating the virtual environment can take several minutes. When the environment is ready, proceed to the next step."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0714a968",
   "metadata": {},
   "source": [
    "## Install packages and load connections\n",
    "\n",
    "This step installs the packages for this notebook and establishes connections to Azure AI Search and Azure OpenAI in Azure AI Foundry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "041e5d89",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install -r requirements.txt --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2df3a118",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dotenv import load_dotenv\n",
    "from azure.identity import DefaultAzureCredential, get_bearer_token_provider\n",
    "import os\n",
    "\n",
    "# Take environment variables from .env\n",
    "load_dotenv(override=True)\n",
    "\n",
    "# This notebook uses the following variables from your .env file\n",
    "search_endpoint = os.environ[\"SEARCH_ENDPOINT\"]\n",
    "credential = DefaultAzureCredential()\n",
    "token_provider = get_bearer_token_provider(credential, \"https://search.azure.com/.default\")\n",
    "aoai_endpoint = os.environ[\"AOAI_ENDPOINT\"]\n",
    "aoai_embedding_model = os.environ[\"AOAI_EMBEDDING_MODEL\"]\n",
    "aoai_embedding_deployment = os.environ[\"AOAI_EMBEDDING_DEPLOYMENT\"]\n",
    "aoai_gpt_model = os.environ[\"AOAI_GPT_MODEL\"]\n",
    "aoai_gpt_deployment = os.environ[\"AOAI_GPT_DEPLOYMENT\"]\n",
    "index_name = os.environ[\"INDEX_NAME\"]\n",
    "knowledge_source_name = os.environ[\"KNOWLEDGE_SOURCE_NAME\"]\n",
    "knowledge_agent_name = os.environ[\"KNOWLEDGE_AGENT_NAME\"]\n",
    "search_api_version = os.environ[\"SEARCH_API_VERSION\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58e8a088",
   "metadata": {},
   "source": [
    "## Create a search index\n",
    "\n",
    "This step creates an index that contains plain text and vector content. You can use an existing index, but it must meet the criteria for [agentic retrieval workloads](https://learn.microsoft.com/azure/search/search-agentic-retrieval-how-to-index). The primary schema requirement is a semantic configuration with a `default_configuration_name`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "ee48bec5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index 'earth-at-night' created or updated successfully.\n"
     ]
    }
   ],
   "source": [
    "from azure.search.documents.indexes.models import SearchIndex, SearchField, VectorSearch, VectorSearchProfile, HnswAlgorithmConfiguration, AzureOpenAIVectorizer, AzureOpenAIVectorizerParameters, SemanticSearch, SemanticConfiguration, SemanticPrioritizedFields, SemanticField\n",
    "from azure.search.documents.indexes import SearchIndexClient\n",
    "from openai import AzureOpenAI\n",
    "from azure.identity import get_bearer_token_provider\n",
    "\n",
    "azure_openai_token_provider = get_bearer_token_provider(credential, \"https://cognitiveservices.azure.com/.default\")\n",
    "index = SearchIndex(\n",
    "    name=index_name,\n",
    "    fields=[\n",
    "        SearchField(name=\"id\", type=\"Edm.String\", key=True, filterable=True, sortable=True, facetable=True),\n",
    "        SearchField(name=\"page_chunk\", type=\"Edm.String\", filterable=False, sortable=False, facetable=False),\n",
    "        SearchField(name=\"page_embedding_text_3_large\", type=\"Collection(Edm.Single)\", stored=False, vector_search_dimensions=3072, vector_search_profile_name=\"hnsw_text_3_large\"),\n",
    "        SearchField(name=\"page_number\", type=\"Edm.Int32\", filterable=True, sortable=True, facetable=True)\n",
    "    ],\n",
    "    vector_search=VectorSearch(\n",
    "        profiles=[VectorSearchProfile(name=\"hnsw_text_3_large\", algorithm_configuration_name=\"alg\", vectorizer_name=\"azure_openai_text_3_large\")],\n",
    "        algorithms=[HnswAlgorithmConfiguration(name=\"alg\")],\n",
    "        vectorizers=[\n",
    "            AzureOpenAIVectorizer(\n",
    "                vectorizer_name=\"azure_openai_text_3_large\",\n",
    "                parameters=AzureOpenAIVectorizerParameters(\n",
    "                    resource_url=aoai_endpoint,\n",
    "                    deployment_name=aoai_embedding_deployment,\n",
    "                    model_name=aoai_embedding_model\n",
    "                )\n",
    "            )\n",
    "        ]\n",
    "    ),\n",
    "    semantic_search=SemanticSearch(\n",
    "        default_configuration_name=\"semantic_config\",\n",
    "        configurations=[\n",
    "            SemanticConfiguration(\n",
    "                name=\"semantic_config\",\n",
    "                prioritized_fields=SemanticPrioritizedFields(\n",
    "                    content_fields=[\n",
    "                        SemanticField(field_name=\"page_chunk\")\n",
    "                    ]\n",
    "                )\n",
    "            )\n",
    "        ]\n",
    "    )\n",
    ")\n",
    "\n",
    "index_client = SearchIndexClient(endpoint=search_endpoint, credential=credential)\n",
    "index_client.create_or_update_index(index)\n",
    "print(f\"Index '{index_name}' created or updated successfully.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39874f61",
   "metadata": {},
   "source": [
    "## Upload sample documents\n",
    "\n",
    "This notebook uses data from NASA's Earth at Night e-book. The data is retrieved from the [azure-search-sample-data](https://github.com/Azure-Samples/azure-search-sample-data) repository on GitHub and passed to the search client for indexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "ded5147b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Documents uploaded to index 'earth-at-night' successfully.\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "from azure.search.documents import SearchIndexingBufferedSender\n",
    "\n",
    "url = \"https://raw.githubusercontent.com/Azure-Samples/azure-search-sample-data/refs/heads/main/nasa-e-book/earth-at-night-json/documents.json\"\n",
    "documents = requests.get(url).json()\n",
    "\n",
    "with SearchIndexingBufferedSender(endpoint=search_endpoint, index_name=index_name, credential=credential) as client:\n",
    "    client.upload_documents(documents=documents)\n",
    "\n",
    "print(f\"Documents uploaded to index '{index_name}' successfully.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0fb9e5f",
   "metadata": {},
   "source": [
    "## Create a knowledge source\n",
    "\n",
    "This step creates a knowledge source that targets the index you previously created. In the next step, you create a knowledge agent that uses the knowledge source to orchestrate agentic retrieval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "e3415954",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Knowledge source 'earth-knowledge-source' created or updated successfully.\n"
     ]
    }
   ],
   "source": [
    "from azure.search.documents.indexes.models import SearchIndexKnowledgeSource, SearchIndexKnowledgeSourceParameters\n",
    "from azure.search.documents.indexes import SearchIndexClient\n",
    "\n",
    "ks = SearchIndexKnowledgeSource(\n",
    "    name=knowledge_source_name,\n",
    "    description=\"Knowledge source for Earth at night data\",\n",
    "    search_index_parameters=SearchIndexKnowledgeSourceParameters(\n",
    "        search_index_name=index_name,\n",
    "        source_data_select=\"id,page_chunk,page_number\",\n",
    "    ),\n",
    ")\n",
    "\n",
    "index_client = SearchIndexClient(endpoint=search_endpoint, credential=credential)\n",
    "index_client.create_or_update_knowledge_source(knowledge_source=ks, api_version=search_api_version)\n",
    "print(f\"Knowledge source '{knowledge_source_name}' created or updated successfully.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c5e0a34",
   "metadata": {},
   "source": [
    "## Create a knowledge agent\n",
    "\n",
    "This step creates a knowledge agent, which acts as a wrapper for your knowledge source and LLM deployment.\n",
    "\n",
    "`EXTRACTIVE_DATA` is the default modality and returns content from your knowledge sources without generative alteration. However, this quickstart uses the `ANSWER_SYNTHESIS` modality for LLM-generated answers that cite the retrieved content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "d3fe4183",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Knowledge agent 'earth-knowledge-agent' created or updated successfully.\n"
     ]
    }
   ],
   "source": [
    "from azure.search.documents.indexes.models import KnowledgeAgent, KnowledgeAgentAzureOpenAIModel, KnowledgeSourceReference, AzureOpenAIVectorizerParameters, KnowledgeAgentOutputConfiguration, KnowledgeAgentOutputConfigurationModality\n",
    "from azure.search.documents.indexes import SearchIndexClient\n",
    "\n",
    "aoai_params = AzureOpenAIVectorizerParameters(\n",
    "    resource_url=aoai_endpoint,\n",
    "    deployment_name=aoai_gpt_deployment,\n",
    "    model_name=aoai_gpt_model,\n",
    ")\n",
    "\n",
    "output_cfg = KnowledgeAgentOutputConfiguration(\n",
    "    modality=KnowledgeAgentOutputConfigurationModality.ANSWER_SYNTHESIS,\n",
    "    include_activity=True,\n",
    ")\n",
    "\n",
    "agent = KnowledgeAgent(\n",
    "    name=knowledge_agent_name,\n",
    "    models=[KnowledgeAgentAzureOpenAIModel(azure_open_ai_parameters=aoai_params)],\n",
    "    knowledge_sources=[\n",
    "        KnowledgeSourceReference(\n",
    "            name=knowledge_source_name,\n",
    "            reranker_threshold=2.5,\n",
    "        )\n",
    "    ],\n",
    "    output_configuration=output_cfg,\n",
    ")\n",
    "\n",
    "index_client = SearchIndexClient(endpoint=search_endpoint, credential=credential)\n",
    "index_client.create_or_update_agent(agent, api_version=search_api_version)\n",
    "print(f\"Knowledge agent '{knowledge_agent_name}' created or updated successfully.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1610cefd",
   "metadata": {},
   "source": [
    "## Set up messages\n",
    "\n",
    "Messages are the input for the retrieval route and contain the conversation history. Each message includes a `role` that indicates its origin, such as `system` or `user`, and `content` in natural language. The LLM you use determines which roles are valid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2ab7b970",
   "metadata": {},
   "outputs": [],
   "source": [
    "instructions = \"\"\"\n",
    "A Q&A agent that can answer questions about the Earth at night.\n",
    "If you don't have the answer, respond with \"I don't know\".\n",
    "\"\"\"\n",
    "\n",
    "messages = [\n",
    "    {\n",
    "        \"role\": \"system\",\n",
    "        \"content\": instructions\n",
    "    }\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4090707f",
   "metadata": {},
   "source": [
    "## Use agentic retrieval to fetch results\n",
    "\n",
    "This step runs the agentic retrieval pipeline to produce a grounded, citation-backed answer. Given the conversation history and retrieval parameters, your knowledge agent:\n",
    "\n",
    "1. Analyzes the entire conversation to infer the user's information need.\n",
    "\n",
    "1. Decomposes the compound query into focused subqueries.\n",
    "\n",
    "1. Runs the subqueries concurrently against your knowledge source.\n",
    "\n",
    "1. Uses semantic ranker to rerank and filter the results.\n",
    "\n",
    "1. Synthesizes the top results into a natural-language answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "918ded26",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieved content from 'earth-knowledge-source' successfully.\n"
     ]
    }
   ],
   "source": [
    "from azure.search.documents.agent import KnowledgeAgentRetrievalClient\n",
    "from azure.search.documents.agent.models import KnowledgeAgentRetrievalRequest, KnowledgeAgentMessage, KnowledgeAgentMessageTextContent, SearchIndexKnowledgeSourceParams\n",
    "\n",
    "agent_client = KnowledgeAgentRetrievalClient(endpoint=search_endpoint, agent_name=knowledge_agent_name, credential=credential)\n",
    "query_1 = \"\"\"\n",
    "    Why do suburban belts display larger December brightening than urban cores even though absolute light levels are higher downtown?\n",
    "    Why is the Phoenix nighttime street grid is so sharply visible from space, whereas large stretches of the interstate between midwestern cities remain comparatively dim?\n",
    "    \"\"\"\n",
    "\n",
    "messages.append({\n",
    "    \"role\": \"user\",\n",
    "    \"content\": query_1\n",
    "})\n",
    "\n",
    "req = KnowledgeAgentRetrievalRequest(\n",
    "    messages=[\n",
    "        KnowledgeAgentMessage(\n",
    "            role=m[\"role\"],\n",
    "            content=[KnowledgeAgentMessageTextContent(text=m[\"content\"])]\n",
    "        ) for m in messages if m[\"role\"] != \"system\"\n",
    "    ],\n",
    "    knowledge_source_params=[\n",
    "        SearchIndexKnowledgeSourceParams(\n",
    "            knowledge_source_name=knowledge_source_name,\n",
    "        )\n",
    "    ]\n",
    ")\n",
    "\n",
    "result = agent_client.retrieve(retrieval_request=req, api_version=search_api_version)\n",
    "print(f\"Retrieved content from '{knowledge_source_name}' successfully.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "886fc687",
   "metadata": {},
   "source": [
    "### Review the retrieval response, activity, and results\n",
    "\n",
    "Because your knowledge agent is configured for answer synthesis, the retrieval response contains the following values:\n",
    "\n",
    "+ `response_content`: An LLM-generated answer to the query that cites the retrieved documents.\n",
    "\n",
    "+ `activity_content`: Detailed planning and execution information, including subqueries, reranking decisions, and intermediate steps.\n",
    "\n",
    "+ `references_content`: Source documents and chunks that contributed to the answer.\n",
    "\n",
    "**Tip:** Retrieval parameters, such as reranker thresholds and knowledge source parameters, influence how aggressively your agent reranks and which sources it queries. Inspect the activity and references to validate grounding and build traceable citations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d4d78fbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "response_contents = []\n",
    "activity_contents = []\n",
    "references_contents = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7fccf4b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "response_content:\n",
      " Suburban belts display larger December brightening than urban cores despite higher absolute light levels downtown because the urban grid encourages outward growth along city borders, facilitated by widespread personal automobile use, leading to extensive suburban and residential municipalities linked by surface streets and freeways. This expansion results in increased lighting in suburban areas during December. The Phoenix nighttime street grid is sharply visible from space due to its regular grid layout of city blocks and streets, with street lighting patterns clearly visible from low-Earth orbit. In contrast, large stretches of interstate highways between Midwestern cities remain comparatively dim because they lack the dense, regular street lighting and urban development seen in Phoenix's metropolitan area. Additionally, brightly lit properties such as large shopping centers, strip malls, and gas stations tend to be located at intersections within the grid, enhancing the visibility of the urban pattern at night. Darker areas like the Phoenix Mountains, agricultural fields, and the Salt River channel contrast with the lit urban grid, further emphasizing the street lighting pattern from space [ref_id:0]. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "# Build simple string values for response_content, activity_content, and references_content\n",
    "\n",
    "# Responses -> Concatenate text/value fields from all response contents\n",
    "response_parts = []\n",
    "if getattr(result, \"response\", None):\n",
    "    for resp in result.response:\n",
    "        for content in getattr(resp, \"content\", []):\n",
    "            text = getattr(content, \"text\", None) or getattr(content, \"value\", None) or str(content)\n",
    "            response_parts.append(text)\n",
    "response_content = \"\\n\\n\".join(response_parts) if response_parts else \"No response found on 'result'\"\n",
    "\n",
    "response_contents.append(response_content)\n",
    "\n",
    "# Print the three string values\n",
    "print(\"response_content:\\n\", response_content, \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7355941b",
   "metadata": {},
   "outputs": [],
   "source": [
    "messages.append({\n",
    "    \"role\": \"assistant\",\n",
    "    \"content\": response_content\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4cef4fd3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "activity_content:\n",
      " [\n",
      "  {\n",
      "    \"id\": 0,\n",
      "    \"type\": \"modelQueryPlanning\",\n",
      "    \"elapsed_ms\": 4225,\n",
      "    \"input_tokens\": 2071,\n",
      "    \"output_tokens\": 125\n",
      "  },\n",
      "  {\n",
      "    \"id\": 1,\n",
      "    \"type\": \"searchIndex\",\n",
      "    \"elapsed_ms\": 443,\n",
      "    \"knowledge_source_name\": \"earth-knowledge-source\",\n",
      "    \"query_time\": \"2025-09-05T19:45:59.785Z\",\n",
      "    \"count\": 0,\n",
      "    \"search_index_arguments\": {\n",
      "      \"search\": \"Reasons for larger December brightening in suburban belts compared to urban cores despite higher absolute light levels downtown\"\n",
      "    }\n",
      "  },\n",
      "  {\n",
      "    \"id\": 2,\n",
      "    \"type\": \"searchIndex\",\n",
      "    \"elapsed_ms\": 274,\n",
      "    \"knowledge_source_name\": \"earth-knowledge-source\",\n",
      "    \"query_time\": \"2025-09-05T19:46:00.060Z\",\n",
      "    \"count\": 1,\n",
      "    \"search_index_arguments\": {\n",
      "      \"search\": \"Factors contributing to the sharp visibility of Phoenix nighttime street grid from space\"\n",
      "    }\n",
      "  },\n",
      "  {\n",
      "    \"id\": 3,\n",
      "    \"type\": \"searchIndex\",\n",
      "    \"elapsed_ms\": 291,\n",
      "    \"knowledge_source_name\": \"earth-knowledge-source\",\n",
      "    \"query_time\": \"2025-09-05T19:46:00.352Z\",\n",
      "    \"count\": 0,\n",
      "    \"search_index_arguments\": {\n",
      "      \"search\": \"Reasons why large stretches of interstate between Midwestern cities appear comparatively dim at night from space\"\n",
      "    }\n",
      "  },\n",
      "  {\n",
      "    \"id\": 4,\n",
      "    \"type\": \"semanticReranker\",\n",
      "    \"input_tokens\": 67716\n",
      "  },\n",
      "  {\n",
      "    \"id\": 5,\n",
      "    \"type\": \"modelAnswerSynthesis\",\n",
      "    \"elapsed_ms\": 3108,\n",
      "    \"input_tokens\": 2806,\n",
      "    \"output_tokens\": 209\n",
      "  }\n",
      "] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Activity -> JSON string of activity as list of dicts\n",
    "if getattr(result, \"activity\", None):\n",
    "    activity_content = json.dumps([a.as_dict() for a in result.activity], indent=2)\n",
    "else:\n",
    "    activity_content = \"No activity found on 'result'\"\n",
    "    \n",
    "activity_contents.append(activity_content)\n",
    "print(\"activity_content:\\n\", activity_content, \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "172df234",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "references_content:\n",
      " [\n",
      "  {\n",
      "    \"type\": \"searchIndex\",\n",
      "    \"id\": \"0\",\n",
      "    \"activity_source\": 2,\n",
      "    \"reranker_score\": 2.5643148,\n",
      "    \"doc_key\": \"earth_at_night_508_page_104_verbalized\"\n",
      "  }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "# References -> JSON string of references as list of dicts\n",
    "if getattr(result, \"references\", None):\n",
    "    references_content = json.dumps([r.as_dict() for r in result.references], indent=2)\n",
    "else:\n",
    "    references_content = \"No references found on 'result'\"\n",
    "    \n",
    "references_contents.append(references_content)\n",
    "print(\"references_content:\\n\", references_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75386ed1",
   "metadata": {},
   "source": [
    "## Continue the conversation\n",
    "\n",
    "This step continues the conversation with your knowledge agent, building upon the previous messages and queries to retrieve relevant information from your knowledge source."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "da260539",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieved content from 'earth-knowledge-source' successfully.\n"
     ]
    }
   ],
   "source": [
    "query_2 = \"How do I find lava at night?\"\n",
    "messages.append({\n",
    "    \"role\": \"user\",\n",
    "    \"content\": query_2\n",
    "})\n",
    "\n",
    "req = KnowledgeAgentRetrievalRequest(\n",
    "    messages=[\n",
    "        KnowledgeAgentMessage(\n",
    "            role=m[\"role\"],\n",
    "            content=[KnowledgeAgentMessageTextContent(text=m[\"content\"])]\n",
    "        ) for m in messages if m[\"role\"] != \"system\"\n",
    "    ],\n",
    "    knowledge_source_params=[\n",
    "        SearchIndexKnowledgeSourceParams(\n",
    "            knowledge_source_name=knowledge_source_name,\n",
    "        )\n",
    "    ]\n",
    ")\n",
    "\n",
    "result = agent_client.retrieve(retrieval_request=req, api_version=search_api_version)\n",
    "print(f\"Retrieved content from '{knowledge_source_name}' successfully.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd1cba0c",
   "metadata": {},
   "source": [
    "### Review the new retrieval response, activity, and results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "35a1bfcb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "response_content:\n",
      " Lava can be found at night by using satellite imagery that captures thermal and infrared wavelengths, which highlight the heat emitted by lava flows. For example, the Landsat 8 satellite combines thermal, shortwave infrared, and near-infrared wavelengths to distinguish very hot lava (bright white), cooling lava (red), and lava flows obscured by clouds (purple) during nighttime observations, as seen in the monitoring of Kilauea's lava flows in Hawaii [ref_id:0]. Similarly, the Operational Land Imager (OLI) and Thermal Infrared Sensor (TIRS) on Landsat 8 can detect the thermal infrared signature of lava flows, such as during Mount Etna's flank eruption in Italy [ref_id:1]. Additionally, the VIIRS Day/Night Band (DNB) on polar-orbiting satellites uses faint light sources like moonlight to detect natural night lights, including glowing lava flows at active volcanoes, enabling their tracking and monitoring for scientific and safety purposes [ref_id:3]. Nighttime satellite images can show the bright glow of lava distinct from city lights, as demonstrated in images of Mount Etna's eruptions [ref_id:2][ref_id:4][ref_id:5]. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "# Build simple string values for response_content, activity_content, and references_content\n",
    "\n",
    "# Responses -> Concatenate text/value fields from all response contents\n",
    "response_parts = []\n",
    "if getattr(result, \"response\", None):\n",
    "    for resp in result.response:\n",
    "        for content in getattr(resp, \"content\", []):\n",
    "            text = getattr(content, \"text\", None) or getattr(content, \"value\", None) or str(content)\n",
    "            response_parts.append(text)\n",
    "response_content = \"\\n\\n\".join(response_parts) if response_parts else \"No response found on 'result'\"\n",
    "\n",
    "response_contents.append(response_content)\n",
    "\n",
    "# Print the three string values\n",
    "print(\"response_content:\\n\", response_content, \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6f74c2c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "activity_content:\n",
      " [\n",
      "  {\n",
      "    \"id\": 0,\n",
      "    \"type\": \"modelQueryPlanning\",\n",
      "    \"elapsed_ms\": 1561,\n",
      "    \"input_tokens\": 2299,\n",
      "    \"output_tokens\": 89\n",
      "  },\n",
      "  {\n",
      "    \"id\": 1,\n",
      "    \"type\": \"searchIndex\",\n",
      "    \"elapsed_ms\": 563,\n",
      "    \"knowledge_source_name\": \"earth-knowledge-source\",\n",
      "    \"query_time\": \"2025-09-05T19:47:37.466Z\",\n",
      "    \"count\": 6,\n",
      "    \"search_index_arguments\": {\n",
      "      \"search\": \"How to locate lava flows at night\"\n",
      "    }\n",
      "  },\n",
      "  {\n",
      "    \"id\": 2,\n",
      "    \"type\": \"searchIndex\",\n",
      "    \"elapsed_ms\": 336,\n",
      "    \"knowledge_source_name\": \"earth-knowledge-source\",\n",
      "    \"query_time\": \"2025-09-05T19:47:37.803Z\",\n",
      "    \"count\": 2,\n",
      "    \"search_index_arguments\": {\n",
      "      \"search\": \"Methods for detecting lava at night\"\n",
      "    }\n",
      "  },\n",
      "  {\n",
      "    \"id\": 3,\n",
      "    \"type\": \"searchIndex\",\n",
      "    \"elapsed_ms\": 463,\n",
      "    \"knowledge_source_name\": \"earth-knowledge-source\",\n",
      "    \"query_time\": \"2025-09-05T19:47:38.267Z\",\n",
      "    \"count\": 0,\n",
      "    \"search_index_arguments\": {\n",
      "      \"search\": \"Techniques for finding lava during nighttime\"\n",
      "    }\n",
      "  },\n",
      "  {\n",
      "    \"id\": 4,\n",
      "    \"type\": \"semanticReranker\",\n",
      "    \"input_tokens\": 68931\n",
      "  },\n",
      "  {\n",
      "    \"id\": 5,\n",
      "    \"type\": \"modelAnswerSynthesis\",\n",
      "    \"elapsed_ms\": 4683,\n",
      "    \"input_tokens\": 5355,\n",
      "    \"output_tokens\": 255\n",
      "  }\n",
      "] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Activity -> JSON string of activity as list of dicts\n",
    "if getattr(result, \"activity\", None):\n",
    "    activity_content = json.dumps([a.as_dict() for a in result.activity], indent=2)\n",
    "else:\n",
    "    activity_content = \"No activity found on 'result'\"\n",
    "    \n",
    "activity_contents.append(activity_content)\n",
    "print(\"activity_content:\\n\", activity_content, \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a6486c8a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "references_content:\n",
      " [\n",
      "  {\n",
      "    \"type\": \"searchIndex\",\n",
      "    \"id\": \"0\",\n",
      "    \"activity_source\": 1,\n",
      "    \"reranker_score\": 2.779123,\n",
      "    \"doc_key\": \"earth_at_night_508_page_60_verbalized\"\n",
      "  },\n",
      "  {\n",
      "    \"type\": \"searchIndex\",\n",
      "    \"id\": \"2\",\n",
      "    \"activity_source\": 1,\n",
      "    \"reranker_score\": 2.7684891,\n",
      "    \"doc_key\": \"earth_at_night_508_page_64_verbalized\"\n",
      "  },\n",
      "  {\n",
      "    \"type\": \"searchIndex\",\n",
      "    \"id\": \"1\",\n",
      "    \"activity_source\": 2,\n",
      "    \"reranker_score\": 2.6940506,\n",
      "    \"doc_key\": \"earth_at_night_508_page_46_verbalized\"\n",
      "  },\n",
      "  {\n",
      "    \"type\": \"searchIndex\",\n",
      "    \"id\": \"4\",\n",
      "    \"activity_source\": 1,\n",
      "    \"reranker_score\": 2.6189032,\n",
      "    \"doc_key\": \"earth_at_night_508_page_66_verbalized\"\n",
      "  },\n",
      "  {\n",
      "    \"type\": \"searchIndex\",\n",
      "    \"id\": \"3\",\n",
      "    \"activity_source\": 2,\n",
      "    \"reranker_score\": 2.5664418,\n",
      "    \"doc_key\": \"earth_at_night_508_page_44_verbalized\"\n",
      "  },\n",
      "  {\n",
      "    \"type\": \"searchIndex\",\n",
      "    \"id\": \"5\",\n",
      "    \"activity_source\": 1,\n",
      "    \"reranker_score\": 2.5382614,\n",
      "    \"doc_key\": \"earth_at_night_508_page_65_verbalized\"\n",
      "  }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "# References -> JSON string of references as list of dicts\n",
    "if getattr(result, \"references\", None):\n",
    "    references_content = json.dumps([r.as_dict() for r in result.references], indent=2)\n",
    "else:\n",
    "    references_content = \"No references found on 'result'\"\n",
    "    \n",
    "references_contents.append(references_content)\n",
    "print(\"references_content:\\n\", references_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e98057c5",
   "metadata": {},
   "source": [
    "## Run an evaluation with Azure AI Foundry\n",
    "\n",
    "To evaluate the groundedness and relevance of the pipeline, run an evaluation with Azure AI Foundry. For more detailed guidance, see [Run evaluations locally by using the Azure AI Foundry SDK](https://learn.microsoft.com/azure/ai-foundry/how-to/develop/evaluate-sdk)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d88117b3",
   "metadata": {},
   "source": [
    "### Prerequisites\n",
    "\n",
    "+ The same [Azure AI Foundry project](ttps://learn.microsoft.com/azure/ai-foundry/how-to/create-projects) you used for agentic retrieval. Set `FOUNDRY_ENDPOINT` to your project endpoint in the `.env` file. You can find this endpoint on the **Overview** page of your project in the [Azure AI Foundry portal](https://ai.azure.com/).\n",
    "\n",
    "+ The `azure-ai-evaluation` package. Run the following command to install it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e2a2777f",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install azure-ai-evaluation --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "80001db4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preparing evaluation data...\n",
      "Starting evaluation...\n",
      "2025-09-05 15:48:04 -0400     620 execution.bulk     INFO     Finished 1 / 2 lines.\n",
      "2025-09-05 15:48:04 -0400     620 execution.bulk     INFO     Average execution time for completed lines: 5.77 seconds. Estimated time for incomplete lines: 5.77 seconds.\n",
      "2025-09-05 15:48:04 -0400     620 execution.bulk     INFO     Finished 2 / 2 lines.\n",
      "2025-09-05 15:48:04 -0400     620 execution.bulk     INFO     Average execution time for completed lines: 2.91 seconds. Estimated time for incomplete lines: 0.0 seconds.\n",
      "======= Run Summary =======\n",
      "\n",
      "Run name: \"relevance_20250905_194758_991736\"\n",
      "Run status: \"Completed\"\n",
      "Start time: \"2025-09-05 19:47:58.991736+00:00\"\n",
      "Duration: \"0:00:06.826371\"\n",
      "\n",
      "2025-09-05 15:48:06 -0400   30648 execution.bulk     INFO     Finished 1 / 2 lines.\n",
      "2025-09-05 15:48:06 -0400   30648 execution.bulk     INFO     Average execution time for completed lines: 7.99 seconds. Estimated time for incomplete lines: 7.99 seconds.\n",
      "2025-09-05 15:48:07 -0400   30648 execution.bulk     INFO     Finished 2 / 2 lines.\n",
      "2025-09-05 15:48:07 -0400   30648 execution.bulk     INFO     Average execution time for completed lines: 4.36 seconds. Estimated time for incomplete lines: 0.0 seconds.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Aggregated metrics for evaluator is not a dictionary will not be logged as metrics\n",
      "Aggregated metrics for evaluator is not a dictionary will not be logged as metrics\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "======= Run Summary =======\n",
      "\n",
      "Run name: \"groundedness_20250905_194758_996238\"\n",
      "Run status: \"Completed\"\n",
      "Start time: \"2025-09-05 19:47:58.996238+00:00\"\n",
      "Duration: \"0:00:08.817964\"\n",
      "\n",
      "======= Combined Run Summary (Per Evaluator) =======\n",
      "\n",
      "{\n",
      "    \"groundedness\": {\n",
      "        \"status\": \"Completed\",\n",
      "        \"duration\": \"0:00:08.817964\",\n",
      "        \"completed_lines\": 2,\n",
      "        \"failed_lines\": 0,\n",
      "        \"log_path\": null\n",
      "    },\n",
      "    \"relevance\": {\n",
      "        \"status\": \"Completed\",\n",
      "        \"duration\": \"0:00:06.826371\",\n",
      "        \"completed_lines\": 2,\n",
      "        \"failed_lines\": 0,\n",
      "        \"log_path\": null\n",
      "    }\n",
      "}\n",
      "\n",
      "====================================================\n",
      "\n",
      "Evaluation complete.\n",
      "For more information, go to the Azure AI Foundry portal.\n"
     ]
    }
   ],
   "source": [
    "# Load connections\n",
    "from dotenv import load_dotenv\n",
    "import os\n",
    "\n",
    "load_dotenv(override=True)\n",
    "\n",
    "foundry_endpoint = os.environ[\"FOUNDRY_ENDPOINT\"]\n",
    "aoai_api_version = os.environ[\"AOAI_API_VERSION\"]\n",
    "\n",
    "# Run the evaluation\n",
    "from azure.ai.evaluation import AzureOpenAIModelConfiguration, GroundednessEvaluator, RelevanceEvaluator, evaluate\n",
    "import json\n",
    "\n",
    "evaluation_data = []\n",
    "print(\"Preparing evaluation data...\")\n",
    "for q, r, g in zip([query_1, query_2], references_contents, response_contents):\n",
    "    evaluation_data.append({\n",
    "        \"query\": q,\n",
    "        \"response\": g,\n",
    "        \"context\": r,\n",
    "    })\n",
    "\n",
    "filename = \"evaluation_data.jsonl\"\n",
    "\n",
    "with open(filename, \"w\") as f:\n",
    "    for item in evaluation_data:\n",
    "        f.write(json.dumps(item) + \"\\n\")\n",
    "\n",
    "model_config = AzureOpenAIModelConfiguration(\n",
    "    azure_endpoint=aoai_endpoint,\n",
    "    api_version=aoai_api_version,\n",
    "    azure_deployment=aoai_gpt_model\n",
    ")\n",
    "\n",
    "# RAG triad metrics\n",
    "groundedness = GroundednessEvaluator(model_config=model_config)\n",
    "relevance = RelevanceEvaluator(model_config=model_config)\n",
    "\n",
    "print(\"Starting evaluation...\")\n",
    "result = evaluate(\n",
    "    data=filename,\n",
    "    evaluators={\n",
    "        \"groundedness\": groundedness,\n",
    "        \"relevance\": relevance,\n",
    "    },\n",
    "    azure_ai_project=foundry_endpoint,\n",
    ")\n",
    "\n",
    "print(\"Evaluation complete.\")\n",
    "studio_url = result.get(\"studio_url\")\n",
    "print(\"For more information, go to the Azure AI Foundry portal.\") if studio_url else None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75777ed2",
   "metadata": {},
   "source": [
    "## Clean up objects and resources\n",
    "\n",
    "If you no longer need Azure AI Search or Azure AI Foundry, delete the resources from your Azure subscription. You can also start over by deleting individual objects."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a14f6fe6",
   "metadata": {},
   "source": [
    "### Delete the knowledge agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "67b6a475",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Knowledge agent 'earth-knowledge-agent' deleted successfully.\n"
     ]
    }
   ],
   "source": [
    "from azure.search.documents.indexes import SearchIndexClient\n",
    "\n",
    "index_client = SearchIndexClient(endpoint=search_endpoint, credential=credential)\n",
    "index_client.delete_agent(knowledge_agent_name)\n",
    "print(f\"Knowledge agent '{knowledge_agent_name}' deleted successfully.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecdfb289",
   "metadata": {},
   "source": [
    "### Delete the knowledge source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b08f5e1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Knowledge source 'earth-knowledge-source' deleted successfully.\n"
     ]
    }
   ],
   "source": [
    "from azure.search.documents.indexes import SearchIndexClient\n",
    "\n",
    "index_client = SearchIndexClient(endpoint=search_endpoint, credential=credential)\n",
    "index_client.delete_knowledge_source(knowledge_source=knowledge_source_name)\n",
    "print(f\"Knowledge source '{knowledge_source_name}' deleted successfully.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a35bfbb1",
   "metadata": {},
   "source": [
    "### Delete the search index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "25f5e6a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index 'earth-at-night' deleted successfully.\n"
     ]
    }
   ],
   "source": [
    "from azure.search.documents.indexes import SearchIndexClient\n",
    "\n",
    "index_client = SearchIndexClient(endpoint=search_endpoint, credential=credential)\n",
    "index_client.delete_index(index_name)\n",
    "print(f\"Index '{index_name}' deleted successfully.\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
